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ABSTRACT 

Designers'  decision  making  in  specifying  adaptive  aiding  systems  is 
considered.  A  study  of  design  decisions  in  specifying  aiding  for  a  fighter  aircraft 
mission  scenario  is  discussed.  Results  indicate  a  high  degree  of  consistency  on  the 
part  of  individual  designers.  However  there  were  substantial  variations  among 
designers  in  terms  of  both  decisions  made  and  information  used  to  make  the 
decisions.  The  implications  of  these  results  for  development  of  design  tools,  as  well 
as  the  types  of  research  studies  whose  results  would  be  valued  by  designers,  are 
considered. 
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INTRODUCTION 

Development  of  mechanisms  to  aid  operators  of  complex  systems  has  a  long 
and  rich  history  (Rouse,  1991).  A  wide  variety  of  approaches  has  been  developed 
for  aiding  operators  in  problem  formulation/stmcturing,  probability  estimation  and 
updating,  selection  among  alternatives,  and  task  execution  and  monitoring.  In 
recent  years,  aiding  has  evolved  to  include  decision  support  systems  and  intelligent 
systems. 

Unfortunately,  despite  the  availability  of  a  growing  number  of  "proof  of 
concept"  studies,  most  aiding  systems  have  been  unsuccessful.  They  have  been 
developed,  but  not  fielded.  Not  infrequently,  they  have  been  fielded,  but  not  used. 

It  can  be  argued  that  this  unfortunate  result  is  due  to  unacceptable  rigidity  in 
the  design  of  aiding  systems  and/or  inflexibility  in  the  aiding  or  automation 
philosophy  underlying  such  systems  (Rouse,  1988).  A  typical  approach  to 
aiding/automation  is  to  computerize  everything  that  can  possibly  be  computerized, 
and  make  whatever  is  left  over  for  humans  as  easy  as  possible.  The  realization  that 
this  approach  often  does  not  work  has  led  to  the  concept  of  adaptive  aiding  systems, 
whereby  the  nature  of  the  aiding,  as  well  as  whether  or  not  the  aiding  is  used,  are 
modified  or  adapted  to  changing  characteristics  of  tasks  and/or  operators'  needs. 

Over  the  past  two  decades,  substantial  conceptual  development  and  a  variety 
of  experimental  efforts  have  proven  the  value  of  the  adaptive  aiding  concept  -  see 
Rouse  (1988)  for  a  review  of  this  work.  These  efforts  have  culminated  in  an  initial 
framework  for  design  of  adaptive  aids,  as  well  as  a  preliminary  set  of  design 
principles  to  guide  design  decisions  (Rouse,  1988, 1991). 

This  design  framework  is  structured  in  terms  of  a  set  of  six  questions: 

What  is  adapted  to? 


2 


NAWCADWAR-92034-60 


•  Who  does  the  adapting? 

•  When  does  adaptation  occur? 

•  What  methods  of  adaptation  apply? 

•  How  is  adaptation  done? 

•  What  is  the  nature  of  communication? 

The  framework  also  includes  alternative  answers  to  these  questions. 

A  primary  difficulty  with  these  questions  and  alternative  answers  is  a  lack  of 
data  upon  which  to  base  choices  among  answers,  as  well  as  specify  details  of 
answers.  The  aforementioned  limited  set  of  design  principles  currently  available  are 
helpful,  but  by  no  means  sufficient  to  cover  the  space  of  possible  answers  to  the  six 
design  questions. 

Initially,  it  might  appear  that  the  obvious  solution  to  this  problem  is  collection 
of  the  requisite  data  to  "fill  in"  the  needed  set  of  principles.  Unfortunately,  the 
experimental  effort  necessary  to  provide  this  data  is  at  least  impractical,  and  possibly 
unimaginable.  The  complexity  of  the  situations  where  adaptive  aiding  is  of  particular 
value  make  the  set  of  potentially  relevant  data  much  too  large  to  imagine  collecting 
it. 


An  alternative  approach  to  this  problem  is  to  shift  the  emphasis  from 
potentially  relevant  data  to  specific  requirements  for  design  decision  making.  In 
other  words,  rather  than  attempt  to  compile  data,  as  well  as  the  resulting  principles, 
to  cover  all  possible  answers  to  the  aforementioned  design  questions,  the  focus 
should  be  on  the  answers  that  designers  typically  pursue  and  the  supporting  data 
that  they  seek  in  the  process. 

In  this  paper,  we  report  the  results  of  pursuing  this  approach.  An 
experimental  investigation  was  performed  involving  designers  whose  task  was  to 


3 


NAWCADWAM2©$4-60 


produce  specifications  for  adaptive  aiding  within  aircraft  mission  scenarios. 
Statistical  methods  were  used  to  identify  relationships  among  a  variety  of  decision 
making  attributes  and  designers'  specification  decisions. 

METHOD 
Aiding  Scenarios 

The  experimental  task  utilized  a  mission  scenario  for  a  2000  +  fighter  aircraft 
involved  in  a  beyond-visual-range  attack  engagement.  This  seven-page  scenario 
was  decomposed  into  42  sce  nario  events,  each  characterized  as  shown  in  Figure  1. 
The  four  elements  of  the  event  descriptions  are  shown  in  Figure  1  and  described 
below. 


First,  the  general  user-system  task  is  shown.  In  this  case,  the  task  was 
judged  to  be  situation  assessment  (SA):  information  seeking.  Events  were 
characterized  in  this  manner  by  two  independent  analysts  using  Rouse's  task 
taxonomy.  This  taxonomy  (Figure  2)  has  been  found  to  be  useful  in  a  variety  of 
efforts  involving  design  of  aiding  systems  for  command  and  control,  nuclear  power, 
manufacturing,  and  design  information  systems  (Rouse,  1986,  1991).  For  the 
purposes  of  this  study,  only  the  main  four  categories  were  employed: 

•  Execution  and  monitoring, 

•  Situation  assessment:  information  seeking, 

•  Situation  assessment:  explanation,  and 

•  Planning  and  commitment. 

Each  of  the  42  scenario  events  were  classified  by  the  two  analysts  in  one  of  these 
categories.  The  small  percentage  of  disagreements  were  resolved  by  discussing  the 
elements  of  the  event  in  question  and  reaching  a  consensus  on  its  classification. 
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The  second  element  of  Figure  1  is  a  prose  description  of  the  event.  This 
information  provides  context,  as  well  as  mission-related  links  to  the  rest  of  the 
scenario.  This  context  is  critical  to  designers  being  able  to  relate  to  the  design  task 
that  they  were  being  asked  to  do. 

The  third  element,  shown  within  a  single  box,  describes  the  foreground  tasks 
for  which  aiding  might  be  specified.  This  information  is  characterized  using  Cohen's 
taxonomy  for  advanced  aircraft  operations  (Cohen,  1990).  This  characterization 
assured  that  all  designers  perceived  the  same  task  requirements. 

The  fourth  and  final  element  of  Figure  1,  shown  within  a  double  box, 
describes  the  background  tasks  that  must  be  performed  despite  the  emergence  of 
new  foreground  demands.  The  distinction  between  foreground  and  background 
tasks  provides  designers  with  the  possibility  of  aiding  new  demands  and/or  ongoing 
demands.  This  distinction  is  important  because  new  demands  can  be  satisfied  by 
either  aiding  these  demands,  or  by  aiding  other  tasks,  thereby  freeing  the  operator  to 
address  the  new  demands. 

The  complete  description  of  all  42  scenario  events,  as  well  as  the 
decomposition  process  used  to  classify  and  characterize  events,  is  provided  in  the 
detailed  technical  report  on  this  effort  (Andes  &  Rouse,  1991).  This  report  also 
describes  in  much  more  detail,  the  experiment,  data  analysis,  and  results  presented 
in  the  remainder  of  this  paper. 

Specification  of  Aiding 

For  each  of  the  42  scenario  events,  designers  were  asked  to  respond  to  the 
multiple  choice  questions  shown  in  Figure  3.  For  the  "motivation”  category, 
designers  were  asked  to  rank  order  the  four  alternatives.  Specifically,  the  most 
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important  reason  for  aiding  was  to  be  ranked  first,  the  next  most  important  reason 
was  to  be  ranked  second,  etc. 

Designers  could  respond  to  the  "tasks  to  be  aided"  category  by  specifying 
neither,  either,  or  both  foreground  and  background  tasks.  As  noted  earlier,  the 
choice  here  concerns  aiding  new  demands  or  aiding  existing  demands  to  enable 
reallocation  of  attention  to  new  demands.  If  designers  specified  both  foreground  and 
background  tasks,  then  two  specification  sheets  were  filled  out  for  the  event,  one  for 
foreground  and  one  for  background  tasks. 

The  "type  of  adaptive  aiding"  category  in  Rgure  3  included  four  possible 
responses  --  three  types  of  aiding  and  a  fourth  choice  of  no  aiding  at  all.  These 
three  types  of  aiding  are  those  postulated  in  the  adaptive  aiding  design  framework 
(Rouse,  1988). 

For  allocation,  the  aiding  system  assigns  task  execution  activities  to  itself. 
Operator  coordination  of  task  performance  is  not  necessary.  While  the  operator 
must  be  notified  of  allocation  recommendations/decisions,  once  the  aid  is  activated, 
it  carries  execution  to  completion. 

With  partitioning,  operator  and  aid  "share"  task  execution.  In  most  cases,  the 
aid  will  indicate  what  it  can  do  (e.g.,  target  designation)  while  the  operator  retains 
remaining  portions  of  tasks  (e.g.,  target  identification).  Partitioning  of  tasks  requires 
that  operator  and  aid  share  information  to  coordinate  task  performance. 

Transformation  involves  modifying  a  (possibly  increasingly)  difficult  task  to 
mitigate  task  demands.  For  example,  an  operator  engaged  in  a  demanding  flight 
control  task  in  conjunction  with  a  difficult  situation  assessment  task  (e.g.,  due  to 
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subsystem  failure)  might  be  aided  by  transforming  flight  control  displays  to  allow  a 
simpler  mode  of  tracking. 

The  "method  of  aid  invocation"  category  in  Figure  3  relates  to  the  intervention 
"threshold"  used  to  activate  aiding.  The  alternative  responses  in  this  category  are 
reasonably  self  explanatory,  with  the  possible  exception  of  the  reference  to  Fitts'  list. 
This  refers  to  the  classic  "men  are  better  at/machines  are  better  at"  lists  that  Fitts 
originated  (Fitts,  1951).  Several  alternative  lists  of  this  type  are  currently  available. 

The  final  category  in  Figure  3,  "operator-aid  communication  requirements," 
refers  to  the  types  of  information  that  operator  and  aid  can  share.  The  three  types  of 
information  include: 

•  Procedural  -  what  the  aid  is  doing. 

•  Product  -  what  the  aid's  outputs  are. 

•  Process  -  how  the  aid  functions. 

Subjects  were  asked  to  respond  to  this  category  by  rating  (0-10)  the  relative  amount 
of  information  needed  of  each  type.  These  three  types  of  information  were  chosen 
based  on  an  analysis  of  information  requirements  for  adaptive  aiding  (Morris,  Rouse, 
&  Ward,  1985).  The  results  of  this  analysis  indicated  that  human  interaction  with 
adaptive  aiding  systems  is  likely  to  be  substantially  affected  by  the  extent  to  which 
procedural,  product,  and  process  information  is  available. 

Decision  Making  Attributes 

in  addition  to  specifying  adaptive  aiding  using  the  categories  in  Figure  3, 
subjects  were  asked  to  rate  (0-10)  the  twelve  attributes  listed  in  Figure  4.  The 
purpose  of  these  ratings  was  to  assess  the  characteristics  of  the  aiding  situation  that 
appeared  to  relate  to  specification  decisions.  Attribute  ratings  were  performed  for 
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each  scenario  event  subsequent  to  completion  of  the  aiding  specification  sheet  for 
that  event.  Subjects  were  asked  to  rate  the  importance  of  each  attribute  to  the 
eventual  success  of  the  specified  aid  (0  «  not  at  ail,  10  =  critical). 


The  twelve  attributes  in  Figure  4  were  defined  as  follows: 

1 .  Anticipated  Aiding  Intervention  Criterion  -  One  of  the  principle  design 
questions  that  the  designer  must  face  is  whether  or  not  to  aid  the 
operator.  Success  of  the  aiding  system  will  greatly  depend  on  what 
criterion  is  used  in  answering:  "Under  what  circumstances  should 
the  aid  intervene?"  There  are  several  criteria  upon  which  aid 
intervention  can  be  based  (e.g.,  unacceptable  operator 
performance,  number  of  concurrent  tasks,  operator  errors).  The 
criterion  must  be  considered  in  the  context  of  aiding.  Within  this 
context,  the  designer  must  also  consider  the  anticipated  knowledge 
representation  of  the  supporting  architecture. 

2.  Tradeoffs  between  cost  of  communication  with  the  operator  about 
error  vs.  aiding  -  In  specifying  aiding  to  assist  the  operator,  for 
example,  when  he  commits  critical  (i.e.,  life  threatening)  errors,  the 
designer  should  consider  several  factors  (e.g.,  time  pressure, 
severity  of  error,  intervention  criterion,  etc.)  in  deciding  whether  to 
communicate  with  the  operator  about  an  error  or  immediately 
activate  aiding  to  compensate  for  the  error. 

3.  Anticipated  difficulty  of  implementing  the  aid  -  Deciding  whether  or 
not  to  aid  the  operator  is  often  influenced  by  how  difficult  the 
implem  ntation  of  such  a  system  may  be.  Additionally,  the  type  of 
aiding  and  interaction  with  the  operator  will  be  affected  by  this 
consideration.  This  attribute  should  be  considered  in  terms  of  the 
level  of  aid  functionality  and  level  of  technology  embedded  in  the 
aiding  system. 

4.  Anticipated  reliability  of  aid  behavior  in  normal  vs.  novel  situations  - 
An  aiding  system  is  only  as  effective  as  designed.  In  this  context, 
reliability  is  defined  as  the  expected,  repeatable  performance  of  the 
aid,  not  mean  time  between  failures  of  the  aid.  The  behavioral 
science  definition  for  reliability  is  used  here  instead  of  the 
engineering  definition.  We  are  more  concerned  with  the  expected 
vs.  actual  behavior  of  the  aid  in  novel  error  situations.  In  other 
words,  can  the  operator  rely  on  the  aid's  functionality  in  novel 
situations? 

5.  Necessary  types  and  level  of  detail  of  operator-aid  communication  - 
In  order  to  facilitate  effective  coordination  between  the  aid  and  the 
operator,  the  aid  must  communicate  useful  information  to  the 
operator.  The  operator  can  receive  information  about  what  the  aid 
is  doing  (procedural),  what  the  aid's  outputs  are  (product),  or  how 
the  aid  is  executing  the  task  (process).  The  designer  should 
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consider  what  information  requirements  the  operator  wilt  have  about 
the  aid  and  the  necessary  detail  of  that  information. 

6.  Overall  risk  (from  design  perspective)  of  aiding  an  event  -  The 
overall  risk  rating  is  of  paramount  importance  to  the  specification  of 
an  adaptive  aiding  system.  Risk  is  defined  as  what  the  designer  is 
willing  to  trade  off  for  potentially  high  functionality.  For  example, 
specifying  an  aid  that  will  intervene  in  critical  error  situations  and 
save  the  operator's  life,  albeit  through  unpredictable  behavior,  may 
be  worth  the  interaction  risk. 

7.  Anticipated  ease  of  aiding  introduction  and  removal  -  The  designer 
must  consider  the  ease  with  which  aiding  can  be  introduced  into  the 
task  environment.  For  example,  will  the  operator  perceive  a  lack  of 
"cognitive  unity”  when  a  task  transformation  is  introduced?  It  is  also 
important  that  tha  negative  cognitive  and  perceptual  effects  of 
removal  of  adaptive  aiding  be  minimized.  In  this  case,  the  designer 
must  consider  the  costs  of  removal  of  aiding  vs.  the  benefit  of 
allowing  the  aid  to  execute  a  task  to  completion. 

8.  Suspected  user  attitude  towards  aiding  -  Some  types  of  aiding  are 
more  acceptable  to  a  user  population  than  others.  If  the  designer  is 
specifying  a  risky  adaptive  aiding  system  from  the  operator's  point  of 
view,  for  example,  the  designer  should  consider  whether  the 
operator  will  want  to  use  it.  The  operator  must  be  (or  become) 
comfortable  with  an  aiding  system  before  he  will  use  it. 

9.  Essential  information  requirements  for  effective  aiding  -  The 
information  requirements,  necessary  to  facilitate  the  aiding  process, 
are  important  considerations  in  specifying  aiding.  Information 
requirements  for  the  operator  about  the  aid,  as  well  as  information 
for  the  aid  about  the  operator,  will  determine  how  and  what  will  be 
aided  in  the  system. 

1 0.  Necessary  level  of  aid  tailorability  -  How  much  of  the  aid's  behavior 
can  (and  should)  be  tailored  based  on  individual  differences  and/or 
population  differences?  This  attribute  affects  aiding  intervention 
thresholds  (e.g.,  "What  is  the  value  that  determines  unacceptable 
performance  for  this  operator?",  etc.).  In  addition,  this  could  pertain 
to  the  level  of  communication  between  the  operator  and  aid  within  a 
particular  task  context). 

11.  Available  technology  to  support  aiding  implementation  -  Even 
though  we  are  analyzing  scenarios  for  Mure  aircraft,  the  designer 
must  consider  what  role  technology  push  and/or  pull  will  play  in 
implementing  some  adaptive  aiding  systems.  Consider  the  range  to 
be  from  none  (all  technology  must  be  developed  to  support  this 
design)  to  all  technology  available  now  "off-the-shelf." 

12.  Number  and  applicability  of  interface/aiding  models  available  - 
Tools,  task  models,  simulations,  etc.  allow  the  designer  to  gain 
insight  into  the  process  that  he  wishes  to  aid.  in  addition, 
embeddable  models  may  facilitate  better  interaction  and  aid 
functionality.  When  specifying  the  aiding  system,  consider  the 
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number  of  available  models,  their  applicability,  and  anticipated 
success  of  using  such  models. 

The  above  definitions  were  provided  to  subjects  prior  to  beginning  the  aiding 
specification  process  and  were  available  for  reference  throughout  the  experiment. 

The  analysis  whereby  the  above  attributes  were  identified  is  presented  in 
Andes  &  Rouse  (1991).  This  analysis  process  involved  reviewing  a  wide  range  of 
attributes  used  by  previous  researchers  and  practitioners.  The  union  of  all  sets  of 
attributes  was  taken  to  form  an  initial  set.  Attributes  were  then  clustered  in  terms  of 
common  orientation  and  purpose.  Redundant  attributes  within  clusters  were  then 
pruned  and  a  consistent  set  of  definitions  chosen. 

Subjects 

Five  subjects  participated  in  this  experiment.  Three  worked  as  individuals 
and  two  worked  as  a  team.  The  team  included  an  adaptive  aiding  system  designer 
and  a  former  U.S.  Air  Force  pilot.  The  reason  for  the  team  was  to  enable 
participation  in  this  experiment  of  an  individual  with  substantial  operational 
experience. 

The  four  adaptive  aiding  analysts  were  all  very  familiar  with  the  concept  of 
adaptive  aiding  and  the  design  framework  discussed  earlier.  Experience  with 
adaptive  aiding  ranged  from  1  to  15  years,  with  an  average  of  7  years. 

Procedure 

Each  subject,  or  team,  performed  independently  in  separate  rooms.  The 
experiment  was  completed  in  one  day,  averaging  5.5  hours  per  subject  or  team. 

There  were  three  segments  to  the  experiment,  run  in  serial  order: 
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1.  Familiarization  with  Context  -  In  the  first  segment,  subjects  were 
asked  to  read  the  textual,  narrative  mission  scenario.  Subjects  were 
requested  to  take  note  of  significant  mission  events,  since  the 
mission  decomposition  was  not  provided  in  the  familiarization  run. 
Note  taking  was  encouraged  to  facilitate  understanding  of  the  event 
sequences  in  the  text. 

2.  Specification  of  Adaptive  Aiding  -  Once  subjects  had  read  the 
scenario  and  understood  the  context,  the  specification  process  was 
begun.  Subjects  were  given  a  segmented  copy  of  the  scenario  just 
read.  Each  of  the  42  segments  were  similar  in  format  to  Figure  1. 
Subjects  were  asked  to  specify  adaptive  aiding  using  specification 
sheets  that  followed  the  format  in  Figure  3. 

3.  Rating  Decision  Making  Attributes  -  Using  the  decision  making 
attributes  listed  in  Figure  4,  subjects  rated  the  importance  of  these 
attributes  to  the  types  of  aiding  specified  for  each  scenario  event. 


In  summary,  subjects  familiarized  themselves  with  the  context  at  the  outset,  and 
then  produced  42  sets  of  specifications  and  ratings,  one  set  per  scenario  event. 


RESULTS 

Basic  Summary  Statistics 

Subjects'  specifications  over  all  42  scenario  events  were  compiled  and 
summary  statistics  calculated.  The  summary  specification  statistics  are  depicted  in 
Figure  5.  This  segment  of  the  analysis  focuses  on  the  most  frequent  responses  by 
each  subject.  The  response  categories  of  primary  interest  were  (abbreviation  in 
parentheses): 

•  Motivation  for  aiding  (Motive), 

•  Tasks  to  be  aided  (Tasks), 

•  Type  of  aiding  (T ype), 

•  Method  of  aid  invocation  (Invocation),  and 

•  Communication  requirements  (Communication). 
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Results  showed  that  two  of  the  subjects  (1  and  2)  based  their  adaptive  aiding 
specifications  primarily  on  operator-related  factors  (i.e.,  workload  increase  as  a 
result  of  task  demands,  performance  degradation  due  to  increased  task  demands, 
and  explicit  user  request  for  aiding),  while  subjects  3  and  4  considered  primarily 
task-related  factors  (i.e.,  implementation  practicality  of  aiding,  tactical  significance  of 
aiding,  allocation  of  task  execution  based  on  the  nature  of  the  tasks  to  be 
conducted).  These  results  suggest  that  subjects  1  and  2  were  "human  activity 
centered,"  while  subjects  3  and  4  were  "task  requirements  centered."  In  other 
words,  the  former  were  more  concerned  with  the  operator's  requirements  necessary 
for  satisfying  the  task  objectives,  while  the  latter  were  more  apt  to  consider  the 
nature  of  the  task  to  be  completed  according  to  mission  requirements. 

It  is  interesting  to  note  that  the  dichotomy  of  human  activity  centered  vs.  task 
requirements  centered  does  not  hold  if  the  type  of  aiding  chosen  is  considered.  As 
shown  in  Rgure  5,  subjects  1  and  2  bracket  subjects  3  and  4  in  terms  of  type  of 
aiding  chosen,  e.g.,  subject  1  chose  "none"  the  least  (5%)  while  subject  2  chose 
"none"  the  most  (43%).  Thus,  the  dichotomy  relates  more  to  why  aiding  is  specified 
rather  than  what  aiding  is  specified.  This  difference  is  further  discussed  in  later 
consideration  of  variations  in  designers'  belief  structures  or  aiding  philosophies. 

The  communication  column  in  Rgure  5  also  illustrates  interesting  contrasts. 
The  average  ratings  for  all  subjects  were  high  for  product  information,  i.e.,  what  the 
aid's  outputs  are.  All  but  one  subject  (no.  3)  gave  low  average  ratings  to  process 
information,  i.e.,  how  the  aid  functions.  For  procedural  information,  i.e.,  what  the  aid 
is  doing,  subjects  1  and  2  both  gave  moderate  average  ratings,  while  subjects  3  and 
4  were  at  opposite  extremes.  Thus,  in  the  communication  category,  subjects  1  and 
2  were,  again,  very  similar  in  all  three  average  ratings.  However,  subjects  3  and  4 
were  only  similar  for  one  of  three  average  ratings. 
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Discriminant  Analyses 

Discriminant  analyses  were  performed  to  determine  the  extent  to  which  the 
type  of  adaptive  aiding  specified  was  related  to  responses  in  the  other  specification 
categories  in  Figure  3.  This  approach  was  taken  because  subjects'  choices  were 
from  categories  rather  than  continuous  response  variables. 

A  discriminant  model  was  constructed  for  each  subject,  or  team,  using  the 
four  response  categories  for  type  of  aiding  as  the  dependent  variable.  There  were 
six  independent  variables,  including  the  responses  to  the  motive,  tasks,  and 
invocation  categories  and  the  ratings  of  requirements  for  procedural,  product,  and 
process  information.  Canonical  coefficients  for  the  resulting  discriminant  functions 
were  computed,  which  enabled  ranking  coefficients,  in  terms  of  absolute  values,  to 
determine  relative  influence. 

The  results  are  shown  in  Figure  6.  As  indicated  by  the  boxed  coefficients  in 
this  figure,  subjects  1  and  2  are  very  similar  in  terms  of  the  factors  that  are  primarily 
associated  with  their  specification  decisions.  Subjects  3  and  4  also  have  a  high 
degree  of  similarity.  These  results  are  consistent  with  the  notions  that  subjects  1 
and  2  were  human  activity  centered,  while  subjects  3  and  4  were  task  requirements 
centered.  More  specifically,  subjects  1  and  2  were  similar  (as  were  subjects  3  and 
4)  in  terms  of  the  variables  they  took  into  account  to  make  decisions.  However,  as 
noted  in  the  discussion  of  Figure  5,  these  pairs  of  subjects  did  not  necessarily  reach 
similar  decisions  for  type  of  aiding. 

Figure  7  indicates  the  goodness  of  fit  of  the  discriminant  models.  Percentage 
agreement  of  predicted  choices  of  types  of  aiding  and  actual  choices  was  60%,  81%, 
74%  and  83%  for  subjects  1-4,  respectively.  The  average  was  75%. 
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Clearly,  the  discriminant  models  match  the  allocation  decisions  better  than 
those  for  partitioning  and  transformation.  Similarly,  the  models  match  partitioning 
decisions  better  than  those  for  transformation.  These  differences  are  probably  due 
to  allocation  being  a  rather  crisp  decision  compared  to  partitioning  and 
transformation.  For  example,  transformation  can  include  many  concepts  for 
modifying  a  task  while  allocation  includes  just  one  concept  -  automation. 

Decision  Making  Attributes 

Subjects'  ratings  of  decision  making  attributes  were  analyzed  in  the  following 
way.  A  mean  rating  for  each  attribute  was  obtained  across  events  for  each  subject. 
To  assure  that  attribute  ratings  were  not  correlated,  the  set  of  ratings  for  each 
subject  were  analyzed  via  Mann-Whitney  pairwise  comparisons.  All  pairwise 
comparisons  for  independence  proved  significant,  rejecting  the  null  hypothesis  that 
the  ratings  were  from  the  same  population.  The  mean  ratings  were  then  normalized 
to  facilitate  comparison  across  subjects.  The  normalized  ratings  were  rank  ordered 
to  determine  the  most  influential  attributes  across  specifications. 

The  top  6  attributes  of  each  subject  were  selected  for  comparison.  After  the 
sixth  attribute,  ratings  tended  to  vary  more  widely.  The  resulting  highly  ranked 
attributes  are  shown  in  Figure  8.  The  rankings  across  subjects,  the  right  column, 
were  compiled  by  ordering  weighted  sums  of  individual  subject’s  rankings.  Due  to 
the  limited  size  of  the  subject  population,  no  attempt  was  made  to  make  statistical 
comparisons  of  rank  orderings. 

Comparing  the  rankings  of  subjects  1  and  2  with  those  of  subjects  3  and  4, 
attributes  4  and  1 1  (i.e.,  reliability  and  availability  of  technology)  were  the  top  two  for 
both  groups  (albeit  in  opposite  order).  Attributes  7,  3,  5,  and  8  (i.e.,  ease  of 
introduction,  difficulty  of  implementation,  oparator-ald  communication,  user  attitude) 
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completed  the  ordering  for  the  first  group,  while  attributes  8,  9,  1,  and  7  (i.e.,  user 
attitude,  information  requirements,  interaction  criterion,  ease  of  introduction) 
completed  the  second  group's  ordering.  Thus,  the  two  groups  were  similar  except 
subjects  1  and  2  emphasized  attributes  3  and  5  (implementation  difficulty  and 
operator-aid  communication)  while  subjects  3  and  4  focused  on  attributes  9  and  1 
(information  requirements  and  intervention  criterion).  Clearly,  the  two  groups  are  not 
as  discriminable  as  they  were  in  earlier  analyses.  This  is  likely  due  to  the  fact  that 
the  types  of  aiding  chosen,  and  hence  the  attributes  of  most  importance,  did  not 
follow  this  dichotomy  of  groups. 

Design  Rule  Elicitation 

In  order  to  gain  further,  albeit  informal,  insight  into  subjects'  decision  making, 
each  subject  was  debriefed  upon  completion  of  the  experiment.  During  this 
debriefing,  subjects  were  queried  about  possible  design  rules  that  may  have 
surfaced  in  the  course  of  the  specification  experiment.  At  least  four  "If-Then"  design 
rules  were  elicited  from  each  subject,  some  with  interesting  implications  for 
generating  aiding  specifications.  The  topics  addressed  ranged  from  the  use  of 
specific  aiding  types  under  certain  conditions  to  how  the  nature  of  operator-aid 
communication  varies  with  the  mission  timeline.  The  complete  set  of  design  rules 
can  be  found  in  Andes  &  Rouse  (1991). 

Most  of  the  rules  were  of  a  general  nature  (e.g.,  IF  pro-occupying  events 
occur,  THEN  aid  background  tasks  according  to  change  In  performance).  These 
rules  not  only  provide  insight  into  a  subject's  design  orientation  and  specification 
strategy,  but  may  also  provide  a  basis  for  eventual  development  of  a  specification 
knowledge  base  to  assist  designers  in  specifying  adaptive  aiding  systems. 
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These  rules  were  also  used  in  a  post-hoc  analysis  of  belief  systems  possibly 
used  by  subjects  during  the  experiment.  These  belief  systems  are  discussed  in  the 
following  section. 

CONCLUSIONS 

The  data  indicate  that  subjects  were  able  to  be  highly  consistent  in  their 
specification  decisions,  particularly  for  allocation  and  partitioning,  but  less  so  for 
transformation.  There  were  also  substantial  differences  among  individuals,  although 
this  was  not  as  pronounced  in  the  analysis  of  decision  making  attributes. 

Designers'  information  needs  for  making  specification  decisions  are 
demonstrated  by  the  results  in  Figures  6  and  8  and  associated  discussions. 
Designers  are  clearly  interested  in  information  about: 

•  Relationships  among  tasks  and  appropriate  types  of  aiding  (Fig.  6), 

•  Appropriate  invocation  criteria  for  different  types  of  aiding  (Fig.  6), 

•  Appropriate  motivations  for  different  types  of  aiding  (Fig.  6), 

•  Anticipated  reliability  of  aid  behavior  in  normal  vs.  novel  situations 
(Fig.  6),  and 

•  Availability  of  technology  to  support  aiding  implementation  (Fig.  8). 

Designers  appear  to  be  much  less  interested  in  information  about: 

•  Tradeoffs  between  costs  of  communicating  vs.  aiding, 

•  Necessary  level  of  aid  tailorabilrty,  and 

•  Number  and  applicability  of  interface/aiding  models  available. 

These  conclusions  would  appear  to  have  important  implications  for  the  types 
of  research  stutfes  whose  results  designers  would  value.  In  particular,  from  Figure 
6  it  can  be  concluded  that  designers  are  likely  to  value  data  that  compare  types  of 
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aiding  and  appropriate  invocation  criteria  as  a  function  of  types  of  tasks  and  the 
motivation  for  aiding  (e.g.,  likely  performance  decrements  vs.  possibly  excessive 
workload).  Further,  based  on  Figure  8  we  can  conclude  that  they  are  concerned  that 
approaches  to  aiding  be  sufficiently  robust  to  be  supportive  in  a  range  of  situations 
and  that  supporting  technology  be  tested  and  practical.  From  this  perspective, 
designers  are  not  likely  to  value  research  results  that  simply  show  that  performance 
is  better  with  aiding  than  without  it  -  they  would  like  to  know  the  specific  ranges  of 
conditions  where  a  particular  type  of  aiding  is  valuable. 

These  conclusions  have  important  implications  for  designing  adaptive  aiding 
systems  and  supporting  the  design  process.  The  summary  statistics,  results  of  the 
discriminant  analyses,  and  the  rank  orderings  of  attribute  ratings  show  what 
information  designers  choose  to  use  in  specifying  aiding.  These  results  also  show 
what  information  they  do  not  use.  Clearly,  a  design  support  environment  should 
provide  what  is  needed  and  wanted,  and  not  burden  the  design  process  with 
additional  information. 

It  is  also  apparent  that  designers  want  specific,  concrete  information  that 
enables  decision  making.  General  principles  are  only  useful  to  the  extent  that  they 
can  be  readily  translated  into  context-specific  decisions.  Thus,  for  example,  "look 
before  you  leap"  is  an  acceptable  general  principle,  but  "look  for  a  50%  increase  in 
response  latency  before  you  automatically  invoke  aiding”  is  a  more  useful  design 
guideline. 

As  a  means  of  integrating  all  of  the  results  presented  in  this  paper,  the 
interpretations  compiled  in  Figure  9  are  offered.  These  interpretations  represent  a 
qualitative  integration  of  all  of  the  statistical  results  presented  earlier,  as  well  as 
designers'  rules  noted  above  and  discussed  in  detail  in  Andes  &  Rouse  (1991).  We 
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speculate  that  these  differences  in  beliefs  underlie  the  individual  differences 
identified  in  the  results.  Further,  the  fact  that  subjects  do  not  neatly  fit  in  just  one  row 
(i.e.,  one  belief  type)  of  Figure  9  may  explain  why  differences  among  groups  did  not 
consistently  emerge.  For  example,  the  agreement  of  subjects  1  and  2  on  why  aiding 
is  needed,  but  their  disagreement  on  what  aiding  is  needed  may,  at  least  In  part,  be 
explained  by  the  interpretations  in  Figure  9.  However,  at  this  point,  we  offer  only  the 
speculation  that  designers'  beliefs  or  aiding  philosophy  (explicit  or  otherwise)  is  likely 
to  affect  their  design  choices  (i.e.,  specifications)  as  well  as  the  information  that  they 
choose  to  employ  in  making  these  choices. 

This  speculation  quite  naturally  raises  another  research  issue  -  what  belief 
system  is  appropriate?  More  specifically,  how  should  designers  think  about  aiding 
decisions?  While  the  "correct"  answer  to  this  question  is  not  clear,  it  is  clear  that  the 
answer  is  likely  to  affect  the  types  of  information  sought  and,  consequently,  the  types 
of  aiding  chosen. 
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sa:  Information  seeking 

|CROWN  provides  as  much  targeting  information  as  possible  as  the  two  forces  dose 
to  about  150  miles.  This  information  is  transmitted  to  the  Blue  Flight's  aircraft  fire 
control  systems  via  secure  link  where  it  appears  on  each  aircraft's  tactical  situation 
display. 


6.0  Intercept 

6.33  Correlate  external  data  with  on-board  data/information 

6.42  Perform  target  acquisition 

6.43  Perform  target  ID 

6.44  Assess  raid 

6.45  Determine  target  assignments 

6.46  Determine  preliminary  targeting 


6.15  Maintain  formation/mutual  support 


Figure  1 .  Scenario  Event  Example 
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Execution  and  Monitoring 

1 .  Implementation  of  Plan 

2.  Observation  of  Consequences 

3.  Evaluation  of  Deviations  from  Expectations 

4.  Selection  Between  Acceptance  and  Rejection 

Situation  Assessment:  Information  Seeking 

5.  Generation/Identification  of  Alternative  Information 
Sources 

6.  Evaluation  of  Alternative  Information  Sources 

7.  Selection  Among  Alternative  Information  Sources 

Situation  Assessment:  Explanation 

8.  Generation  of  Alternative  Explanations 

9.  Evaluation  of  Alternative  Explanations 

10.  Selection  Among  Alternative  Explanations 

Planning  and  Commitment 

1 1 .  Generation  of  Alternative  Courses  of  Action 

12.  Evaluation  of  Alternative  Courses  of  Action 

13.  Selection  Among  Alternative  Courses  of  Action 


Figure  2.  Taxonomy  of  User-System  Tasks 
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Motivation  for  Specifying  Adaptive  Aiding 

-  Estimated  performance  degradation  without  aiding 

-  Projected  workload  in  the  scenario  event 

-  Tactical  significance  of  scenario  event 

-  Projected  implementation  practicality 

Tasks  to  be  Aided 

-  Foreground 
*  Background 

Type  of  Adaptive  Aiding 

-  Allocation 

-  Partitioning 

-  Transformation 

-  None 

Method  of  Aid  Invocation 

-  Unacceptable  system  performance 

-  Number  of  concurrent  operator  tasks 

-  Operator  resource  allocation  estimate 

-  Nature  of  task  (Fitts’  list  reference) 

-  Other  (e.g.,  operator  requests  aiding) 

Operator-Aid  Communication  Requirements 

-  Procedural 

-  Product 

-  Process 


Figure  3.  Specification  Categories 
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1 .  Anticipated  aiding  intervention  criterion 

2.  Tradeoffs  between  cost  of  communicating  with  operator 
about  error  vs.  aiding 

3.  Anticipated  difficulty  of  implementing  the  aid 

4.  Anticipated  reliability  of  aid  behavior  in  normal  vs.  novel 
situations 

5.  Necessary  types  and  level  of  detail  of  operator-aid 
communication 

6.  Overall  risk  (from  design  perspective)  of  aiding  this 
event 

7.  Anticipated  ease  of  aiding  introduction  and  removal 

8.  Suspected  user  attitude  towards  aiding 

9.  Essential  information  requirements  for  effective  aiding 

1 0.  Necessary  level  of  aid  tailorability 

1 1 .  Availability  of  technology  to  support  aiding 
implementation 

1 2.  Number  and  applicability  of  interface/aiding  models 
available 

Figure  4.  Decision  Making  Attributes 
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Allocation 

Subject  1 

tasks 

(1.018) 

prod 

(1.015) 

proeed 

(0.314) 

R2  0.886 

Subject  2 

tasks 

(2.780) 

proeed 

(-1.228) 

prod 

(1.010) 

process 

'0.867) 

Invoc 

(0.556) 

R2  0.998 

Subject  3  tasks  (0.568) 


2  0.910 


Subject  4  tasks  (0.730) 
Invoc  (0.585) 


Note:  coefficients  <13 1  excluded 


Partitioning 


Invoc  (0.867) 
process  (0.747) 
proeed  (-0.614) 
product  (-0.369) 
2  |  0.348 


Invoc  (-0.960) 
proeed  (-0.71 8) 
tasks  (0.551) 
motive  (0.396) 


Transformation 

motive 

]  (0.742) 

tasks 

(0.441) 

proeed 

(-0.401) 

Invoc 

(-0.383) 

motive  I  (-1.016) 
proeed  (-0.871) 
tasks  (0.657) 


2  I  0.238 


(0.568) 

motive  (1.103) 

proeed  (0.767) 

(0.527) 

tasks  (-0.762) 

Invoc  (0.588) 

product  (0.395) 

tasks  (-0.546) 

proeed  (0.315) 

process  (0.534) 

motive  (-0.335) 

proceed  (0.798) 
tasks  (-0.563) 

motive  (0.544) 
process  (0.506) 


2  0.594 


product  (-0.914) 
Invoc  (0.569) 
tasks  (0.554) 
motive  (-0.520) 


Figure  6.  Discriminant  Coefficients  and  R  by  Subject 
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Human  Task 
Activity  Requirements 
Centered  Centered 
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Belief 

Type  of  Aiding 

Allocation 

(A) 

Partitioning 

(p> 

Transformation 

(T) 

Type 

1 

Let  aid 
execute  well- 
defined  task. 

(SI,  S3.  S4) 

Aid  what  tasks 
pilot  cannot 
attend  to  in 
complex  task. 

(S2,  S4) 

Transform 
difficult  manual 
task. 

(S3,  S4) 

II 

Only  allocate 
task  when 
operator 
cannot 
attend  to  it. 

(S2) 

Leave  ill-defined 
parts  of  complex 
task  to  human, 
aid  all  else. 

(SI,  S3) 

Transform 
difficult  situation 
assessment  or 
planning  task. 

(SI,  S2) 

Note:  Parentheses  Indicate  subjects. 

Figure  9.  Alternative  Belief  Structures  Influencing  Specification  Strategies 
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